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Abstract 


The  Institute  for  Information  Technology  Applications’  Operational  Test  and 
Evaluation  Support  Tool  (OPS  TEST)  is  an  automated  testbank  and  on-line 
testing  capability  designed  to  support  the  testing  requirements  of  operational 
Department  of  Defense  units  with  training  and  standardization/evaluation 
testing.  OPS  TEST  on-line  testing  supports  a  variety  of  levels  of  testing  from 
formal,  highly  controlled  certification  testing  to  test-taker  created  practice 
tests.  Test  items  are  stored  as  Microsoft  Word  documents  and  can  include  a 
wide  variety  of  embedded  graphics,  pictures,  and  charts.  In  addition,  test 
items  may  include  associated  multimedia  clips  that  the  test-taker  can  play  as 
part  of  the  test  item.  Feedback  mechanisms  to  the  test-taker  include  textual 
messages  as  well  as  hyperlinks  to  associated  reference  material  on  the 
Internet,  intranet,  or  shared  network  file  systems.  Test  administration 
functions  within  OPS  TEST  include  the  ability  to  add  and  maintain  test  items, 
build  and  maintain  master  and  specific  tests,  track  test  results,  and  analyze 
the  performance  of  test  items.  The  OPS  TEST  software  and  associated 
documentation  as  well  as  some  technical  support  are  available  from  IITA  to 
operational  DoD  units  upon  request. 


Executive  Summary  of  OPS  TEST 


What  can  OPS  TEST  do  to  support  the  operational  unit’s  training 
and  standardization/evaluation  functions? 


Purpose  of  this  document 


The  United  States  Air  Force  Academy’s  Institute  for  Information  Technology  Applications 
(IITA),  in  concert  with  U.S.  Space  Command’s  Cheyenne  Mountain  Operations  Center 
(CMOC),  has  developed  an  automated  testing  environment  to  support  the  operational 
training  and  standardization/evaluation  testing  of  the  CMOC  operations  crew-members. 
This  capability  has  been  adapted  to  an  easily  configurable  implementation  that  other 
Department  of  Defense  (DoD)  operational  units  can  use  to  support  their  training  and 
standardization/evaluation  testing.  We  designed  this  document  to  provide  both  command 
and  unit  level  leadership  with  a  description  of  what  this  automated  testing  environment 
may  be  able  to  do  to  support  their  missions. 


Overview  of  OPS  TEST’S  capabilities 


Many,  if  not  most,  Department  of  Defense  (DoD)  operational  units  have  both  training  and 
standardization/evaluation  (Stan/Eval)  functional  units  that  respectively  facilitate  both  initial 
and  recurring  training  of  operational  personnel  and  establish,  enforce,  and  evaluate 
standards  of  performance.  Both  the  training  and  evaluation  functions  frequently  make  use 
of  written  testing  as  well  as  hands-on  assessments.  IITA’s  Operational  Test  and 
Evaluation  Support  Tool  (OPS  TEST)  provides  automated  support  for  all  aspects  of  what 
would  normally  be  the  written  testing  required  for  both  training  and  evaluation. 

OPS  TEST  Test-Taking 

With  OPS  TEST,  all  test-taking  is  accomplished  interactively  on  the  computer  with 
automatic  recording  of  test  results.  Specific  test-taking  features  include  support  for  formal, 
password-protected  certification  tests;  less  formal  update  training  tests;  as  well  as  test- 
taker  generated  practice  tests  on  topics  of  their  own  selection.  Individual  test  items  are 
multiple-choice  or  true/false  in  form  and  may  include  graphics,  tables,  and  plots. 
Furthermore,  multimedia  clips  using  audio  and/or  video  can  be  associated  with  test  items 
and  played  by  the  test-takers  as  part  of  the  test  item.  Feedback  mechanisms  provide  the 
test-taker  with  instructional  and  reference  material  on  any  items  that  they  may  have 
answered  incorrectly.  One  of  these  feedback  mechanisms  is  a  hyperlink  that  can  be  set  to 
provide  access  to  either  Internet  or  intranet  web  sites  or  to  any  locally  available  electronic 
document.  Additional  test-taking  features  provide  options  for  allowing/disallowing  re¬ 
entrance  to  the  test,  random  reordering  of  test  items,  and  dynamic  creation.  Test-takers 
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can  move  about  freely  within  formal  tests  with  special  features  that  take  them  to  test  items 
that  they  have  not  yet  answered.  Scores  and  feedback  on  missed  items  are  provided  at 
the  end  of  these  formal  tests.  For  test-taker  generated  practice  tests,  the  feedback  is 
immediate  for  each  test  item.  The  form  and  button  driven  test-taking  component  has 
proven  very  easy  to  use  by  all  test  takers  and  has  significantly  reduced  the  test  monitoring 
and  feedback  delivery  duties  of  the  test  administrators. 

OPS  TEST  Testbank  Administration 

For  the  personnel  assigned  to  the  training  and  standardization/evaluation  functional  units, 
OPS  TEST  provides  a  full  suite  of  support  functions  that  can  significantly  reduce  their 
administrative  burdens  while  assisting  them  to  evaluate  the  effectiveness  their  test  items. 
These  administrative  functions  are  organized  under  four  functional  areas:  test  item 
maintenance,  test  building  and  maintenance,  test  results  tracking,  and  test  item  analysis. 

For  test  item  maintenance,  individual  test  items  are  held  as  Microsoft  Word  documents 
and  are  easily  entered  and  updated  using  the  Word  program  that  is  integrated  into  test 
item  maintenance  form.  A  comprehensive  set  of  reference  data  is  kept  with  each  test  item 
for  ease  of  locating  desired  items. 

OPS  TEST’S  test  building  and  maintenance  functions  are  organized  around  the  concept  of 
master  tests,  typically  designed  for  a  functional  position,  and  specific  tests  for  individuals 
that  are  instances  of  the  master  tests.  Form-driven  query  mechanisms  allow  test  items  to 
be  easily  located  for  inclusion  in  the  test.  Test  summaries  provide  efficient  mechanisms 
for  assessing  proper  coverage  of  topics  and  overall  organization  of  the  test. 

Test  administrators  can  track  the  results  and  status  of  tests  in  a  variety  of  ways.  Test 
status  and  results  can  be  selectively  displayed  via  a  variety  of  selection  criteria  including 
status  of  the  test,  the  master  test,  the  test-taker,  the  test  administrator,  and/or  a  date  range 
for  test  completion.  The  selected  test  results  are  shown  in  a  summary  form  that  includes 
the  test,  test-taker,  score,  date  completed,  and  suspense  date  for  each  selected  test.  The 
administrator  can  print  test  results  for  a  particular  test  or  for  the  summary  of  the  tests. 
Furthermore,  administrators  can  view  and  update  any  particular  test  on-line  via  a  button 
provided  with  each  selected  test. 

The  fourth  set  of  functions  provided  by  OPS  TEST  support  the  analysis  of  test  items.  With 
these  functions,  the  administrator  can  select  a  set  of  test  items  for  analysis  via  a  wide 
variety  of  criteria.  For  each  selected  test  item,  a  summary  of  the  item’s  performance  is 
provided  including  the  item’s  overall  ease  (percentage  of  times  it  was  correctly  answered) 
and  a  distribution  of  which  alternative  responses  were  chosen.  This  information  is 
provided  for  all  usages  as  well  as  grouped  by  crew  positions  and  by  individual  test-takers. 

More  Details  on  OPS  TESTs  Capabilities 

More  details  on  OPS  TEST’S  capabilities  can  be  found  in  Chapter  2,  which  provides 
operational  use  scenarios  with  sample  screen  images,  and  Chapter  4,  which  provides  a 
comprehensive  listing  of  capabilities.  Chapter  3  summaries  the  history  behind  OPS  TEST 
and  it’s  development  and  implementation  in  support  of  the  Cheyenne  Mountain 
Operations  Center. 
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What’s  involved  in  adopting  OPS  TEST? 


If  a  particular  operational  unit  would  like  to  adopt  OPS  TEST  to  support  its  operational 
testing,  they  should  contact  one  of  the  IITA  personnel  listed  in  the  following  section.  IITA 
can  provide  the  OPS  TEST  software,  user  and  support  documentation,  and  limited 
telephone  and/or  on-site  technical  support  (with  TDYs  normally  funded  by  the  adopting 
unit).  The  technical  support  can  assist  the  adopting  unit  with  installation  of  OPS  TEST  as 
well  as  the  possible  formulation  of  procedures  to  migrate  existing  test  items  into  the  OPS 
TEST  database.  IITA  can  also  provide  some  initial  training  on  both  test-taking  and  test 
administration.  More  details  on  what  IITA  can  provide  are  presented  in  Chapter  5. 

In  addition  to  the  OPS  TEST  software,  IITA  can  provide  test-writing  workshops  given  by 
their  US  Air  Force  Academy  experts  in  the  area  of  assessment  instrument  development. 
These  workshops  typically  take  place  at  the  adopting  unit’s  facility  with  the  necessary 
instructor’s  travel  expenses  funded  by  the  adopting  unit. 

The  adopting  unit  will  need  to  provide  the  hardware  and  software  infrastructure  required 
by  OPS  TEST.  Typically,  this  is  a  local  area  network  with  a  shared  network  disk  drive  and 
Windows-based  personal  computers.  If  necessary,  OPS  TEST  can  also  be  implemented 
on  a  single,  stand-alone  personal  computer.  OPS  TEST  runs  as  an  Office  97  Access 
application  so  the  adopting  unit  would  preferably  have  Microsoft’s  Office  97  (Professional 
Edition)  suite  of  programs  installed  on  their  computers,  but  only  Microsoft  Word  97  is 
required.  An  Office  2000  version  can  also  be  provided  if  necessary. 

Personnel  from  the  adopting  unit  would  be  responsible  for  installing  and  configuring  the 
OPS  TEST  software  as  well  as  migrating  existing  test  items  into  the  OPS  TEST  database, 
but  technical  assistance  is  available  from  IITA  for  all  of  these  tasks.  On-going 
maintenance  of  the  test  items  and  tests  is  the  responsibility  of  the  adopting  unit. 


OPS  TEST  points  of  contact 


Further  information  on  OPS  TEST  can  be  obtained  from  the  following  contacts  at  the 
Institute  for  Information  Technology  Applications  (IITA): 

•  General  Requests:  Current  Managing  Director  of  IITA,  (719)  333-6748,  DSN: 
333-3978,  email:  iita@usafa.af.mil 

•  OPS  TEST  Project  Leader:  Dr.  Steve  Hadfield,  (719)  333-7474,  DSN:  333- 
7474,  email:  steve.hadfield@usafa.af.mil 

•  Educational  Information  Technologies  Director:  Dr  Eric  Hamilton,  (719)  333- 
8325,  DSN  333-8325,  email:  eric.hamilton@usafa.af.mil 

Or  via  mail  to:  IITA,  HQ  USAFA/DFE,  Suite  4K25 

USAF  Academy,  CO  80840-6200 
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Chapter 


Operational  Use  of  OPS  TEST 

How  would  an  operational  unit  make  use  of  OPS  TEST? 


This  section  describes  use  of  OPS  TEST  at  a  hypothetical  Space  Command  unit  and  is 
intended  to  illustrate  how  OPS  TEST  might  integrate  into  and  support  the  operations  of 
such  a  unit. 


Test-taking  with  OPS  TEST 


In  June,  2nd  Lt  Jeff  Jones  arrives  at  his  new  unit,  the  99th  Space  Squadron.  After  initial  in¬ 
processing,  he  begins  a  rigorous  training  program  that  consists  of  hands-on  operation  of 
the  mission  computer  systems,  numerous  demonstrations  and  exercise  scenarios,  and 
tons  of  reading  including  operating  instructions,  regulations,  and  technical  manuals. 

Lt  Jones  progresses  quickly  and  now  wishes  to  self-assess  his  level  of  knowledge  in  key 
areas.  He  goes  to  one  of  the  unit’s  personal  computers  on  their  local  area  network  (LAN) 
and  starts  up  the  OPS  TEST  Test-Taking  software.  At  the  initial  menu,  he  selects  the 
“Build  Practice  Test  from  the  MQF”  button  (“MQF”  stands  for  Master  Question  File).  At 
this  point,  the  OPS  TEST  test-taking  software  presents  him  with  a  simple  query  form  that 
he  can  use  to  select  items  on  any  combination  of  selection  criteria.  There  are  pull-down 
menus  for  each  selection  criteria,  which  OPS  TEST  creates  automatically  based  on  what 
was  entered  for  each  test  item  by  the  OPS  TEST  administrators.  Furthermore,  he  can  just 
place  his  cursor  over  a  selection  criteria  field  and  a  caption  comes  up  that  gives  more 
information  about  that  field.  For  most  fields,  he  can  also  use  wildcards  such  as  “99SS  01*” 
with  the  “*”  replacing  any  combination  of  characters  so  he  can  get  all  the  MQF  questions 
based  on  the  99th  Space  Squadron’s  Operating  Instructions  (i.e.  99SS  01  1-1,  99SS  01  2- 
3,...). 

He  makes  his  selection  and  clicks  the  “Take  Practice  Test”  button  to  begin  working 
through  the  test  items.  He’s  impressed  to  find  out  that  many  of  the  test  items  have  graphs, 
pictures,  and  mission  screen  images  integrated  directly  into  the  test  item.  Furthermore, 
some  test  items  even  have  multimedia  clips  associated  with  them.  He  just  needs  to  click 
on  the  “Play  Multimedia”  button  to  have  the  clip  run.  The  first  such  test  item  has  a  video 
clip  of  a  space  shuttle  lift-off. 

Since  the  test  doesn’t  count,  he  purposely  answers  incorrectly  to  see  what  happens. 
Once  he  does  this,  OPS  TEST  tells  him  that  he  answered  incorrectly  and  shows  him  both 
his  answer  and  the  correct  one.  In  addition  to  that,  OPS  TEST  provides  a  short  text 
message  explaining  the  reasoning  behind  the  correct  answer,  a  reference  on  that  topic, 
and  a  hyperlink  to  an  Internet  web  page  that  has  plenty  of  information  on  the  Space 
Transportation  System.  Lt  Jones  thinks  to  himself  that  this  is  an  enjoyable  way  to  learn. 
On  the  next  page  is  a  sample  of  the  screens  that  Lt  Jones  encountered  during  his  MQF 
self-assessment  practice  test. 
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He  wonders  whether  working  through  all  of  the  MQF  items  will  allow  him  to  preview  his 
training  and  stan/eval  test  items.  He  is  somewhat  disappointed  to  learn  that  OPS  TEST 
separates  items  by  a  “category”  field  and  that  he  can  only  get  at  the  items  from  the  MQF 
category  and  not  from  the  training  or  evaluation  categories.  (Note:  The  test  administrators 
may  put  MQF  items  in  the  training  and  evaluation  tests  if  they  wish). 

As  Lt  Jones  moves  further  along  in  his  training,  he  informs  his  trainer,  Capt  Cindy  Smith, 
that  he's  ready  to  work  through  the  first  of  his  training  tests.  These  are  standardized  tests 
of  50  items  each  that  are  built  on  demand.  To  ensure  they  are  secure,  Capt  Smith  has 
placed  a  distinct  password  on  each  of  them.  She  can  periodically  change  these 
passwords  to  protect  against  the  tests  being  compromised.  To  further  protect  against 
possible  compromise  of  the  tests,  Capt  Smith  has  checked  the  box  to  have  the  test  items 
randomly  reordered  each  time  the  test  is  taken.  Since  they  are  training  tests,  Capt  Smith 
has  specified  that  the  tests  are  “re-entrant”,  meaning  the  test-taker  can  work  through  part 
of  them,  exit  OPS  TEST  Test-Taking,  and  then  come  back  to  the  test.  Capt  Smith  likes 
this  feature  since  she’s  purposely  designed  the  tests  to  force  the  trainee  to  dig  through  the 
manuals.  In  fact,  she  creates  similar  monthly  training  tests  for  her  already  certified  crew 
members  to  refresh  themselves  on  key  areas  and  new  procedures. 

Lt  Jones  accesses  the  training  test  via  the  “Take  Monthly  Training  Test”  button  on  the 
main  OPS  TEST  Test-Taking  menu.  The  screen  he  gets  has  him  specify  his  name,  the 
test  he  wants  to  take,  and  the  password  for  that  test.  In  addition,  some  other  fields  will  let 
him  get  back  into  the  test  later.  The  test  items  in  this  test  are  presented  a  bit  differently; 
they  are  not  immediately  scored  when  he  selects  an  answer.  Instead,  he  can  move 
forward  and  backward  within  the  test  either  to  the  previous  or  next  items  or  to  the  previous 
or  next  un-answered  item.  When  he  thinks  he  has  finished  the  test,  he  clicks  the  “Grade 
Test”  button.  OPS  TEST  Test-Taking  tells  him  that  he  neglected  to  answer  four  test  items 
and  provides  him  the  opportunity  to  go  back  and  answer  them  prior  to  having  the  test 
graded.  Wisely,  he  chooses  to  do  this.  Now  with  all  the  test  items  answered,  he  clicks  the 
“Grade  Test”  button  again.  His  overall  score  is  reported  and  he  is  given  a  chance  to  go 
back  and  review  the  three  test  items  that  he  missed.  Like  the  MQF  items,  he  gets  some 
feedback  text,  an  associated  reference,  and  a  hyperlink  that  this  time  takes  him  to  some 
relevant  operating  instructions  stored  as  Word  and  Adobe  .pdf  files  on  the  unit’s  shared 
network  disk  drive. 

Once  he’s  passed  all  his  training  tests  and  other  training  activities,  Capt  Smith 
congratulates  Lt  Jones  and  passes  him  off  to  Major  Mike  Knailem,  the  Chief  of  the 
Standardization  and  Evaluation  Branch  for  formal  certification.  Lt  Jones  passes  his 
grueling  hands-on  evaluation  and  must  now  pass  the  written  test  for  full  certification  to  join 
an  operational  crew.  Major  Knailem  uses  one  of  his  two  standardized  tests  for  Lt  Jones’ 
position.  This  test  is  specifically  built  for  Lt  Jones  with  a  unique  password,  which  Major 
Knailem  specifies  as  “uFAILtoad”  using  his  normally  dry  sense  of  humor.  By  default,  this 
test  cannot  be  re-entered  and  Lt  Jones  must  take  the  entire  test  in  one  sitting  and  cannot 
use  any  notes  or  references.  To  ensure  compliance,  Major  Knailem  makes  Lt  Jones  take 
the  test  on  the  computer  in  the  Stan/Eval  office.  Being  brilliantly  trained  by  Capt  Smith  and 
having  made  extensive  use  of  the  MQF  via  OPS  TEST  for  self-assessment,  Lt  Jones 
passes  his  evaluation  test  with  a  perfect  score.  However,  Lt  Jones  realizes  that  he  must 
also  pass  periodic  training  tests  from  Capt  Smith  and  yearly  evaluation  tests  from  Major 
Knailem,  so  he’s  going  to  continue  to  use  OPS  TEST  Test-Taking  with  the  MQF  to  keep 
his  knowledge  base  current. 
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Test  administration  with  OPS  TEST 


Both  Capt  Smith,  from  the  Training  Branch,  and  Major  Knailem,  from  the  Standardization 
and  Evaluation  Branch,  can  also  make  use  of  the  administrative  components  of  OPS 
TEST.  In  fact,  they  each  have  slightly  different  versions  of  OPS  TEST  Administration. 
Capt  Smith  has  the  training  version  which  only  allows  access  to  test  items  whose 
“Category”  does  not  start  with  “Eval”.  That  is,  she  can  not  see  any  of  Major  Knailem’s 
evaluation  test  items,  but  she  can  access  and  update  all  of  her  training  items  as  well  as 
the  MQF  items.  Major  Knailem  has  the  evaluation  version,  which  allows  him  access  to  all 
test  items  in  the  OPS  TEST  database.  Hence,  he  can  use  any  test  items  that  he  wants  in 
his  evaluation  tests,  including  MQF  items. 

Other  than  the  set  of  test  items  and  tests  that  can  be  accessed,  the  two  versions  of  OPS 
TEST  Administration  are  identical.  Each  provides  four  primary  functions  from  the  main 
menu  which  are  Test  Item  Maintenance,  Build/Maintain  Tests,  View  Test  Results,  and 
Test  Item  Analysis.  To  insure  that  the  OPS  TEST  Administration  capabilities  are  secure, 
each  installation  has  a  master  password  that  protects  access  to  the  OPS  TEST 
Administration  functions.  This  password  can  be  easily  changed  by  the  Security  option  on 
the  main  menu  bar.  In  addition,  OPS  TEST  Administration  provides  a  Configuration 
option  on  the  main  menu  bar  that  allows  administrators  to  update  the  codes  and 
descriptions  used  for  classifying  operational  tasks  that  are  testable,  information  on 
individual  crew  members,  and  operational  centers  and  crew  positions.  They  can  also 
update  display  parameters  like  the  form  labels,  the  system-high  classification,  the  name  of 
the  task  codes,  and  the  introductory  message  shown  on  the  initial  menus  of  both  OPS 
TEST  Test-Taking  and  Administration. 

Returning  to  the  four  primary  functions,  Capt  Smith  has  found  that  one  of  her  training  test 
items  has  become  out-dated  due  to  a  recent  change  in  procedures.  She  clicks  the 
“Maintain  Test  Items”  button  and  gets  a  query  form  that  she  can  use  to  locate  the 
particular  test  item.  As  with  all  the  other  OPS  TEST  query  forms,  most  all  of  the  pull- 
downs  are  built  dynamically  with  the  data  entered  into  the  OPS  TEST  database  and 
wildcards  can  be  used  to  broaden  the  search  criteria.  After  specifying  the  search 
parameters,  she  clicks  on  the  “Retrieve  Selected  Test  Items”  button.  Four  test  items  are 
returned  and  Capt  Smith  easily  moves  through  them  to  find  the  item  she  needs  to  update. 
To  change  the  test  item,  she  simply  double-clicks  within  the  test  item  displayed  on  the 
form  and  the  test  item  is  brought  up  with  Microsoft  Word  all  within  the  field  of  the  OPS 
TEST  Test  Item  Maintenance  form.  At  this  point  Capt  Smith  has  all  the  capabilities  of 
Word  available  to  her  and  can  even  embed  any  type  of  graphics  or  pictures  that  she 
wishes. 

Also  on  the  Test  Item  Maintenance  form  are  several  reference  fields  with  appropriate  pull- 
downs.  There  are  also  several  buttons  at  the  bottom  of  the  form  with  small  pictures 
(icons)  on  them.  By  positioning  her  cursor  over  the  button,  a  small  caption  box  appears 
that  describes  the  function  of  that  button.  The  first  button,  a  downward  pointing  hand, 
takes  her  to  a  second  page  where  she  can  specify  feedback  text  to  be  displayed  if  the  test 
item  is  incorrectly  answered.  There’s  also  a  place  for  specifying  a  hyperlink  to  an 
associated  reference  which  can  be  either  a  Internet  or  intranet  web  site  or  any  document 
on  the  unit’s  LAN.  These  could  be  things  like  Word  documents,  Excel  spreadsheets,  or 
Adobe  Acrobat  portable  document  format  (.pdf)  files.  In  addition,  there  is  a  small  box  that 
can  hold  an  associated  multimedia  clip,  together  with  buttons  that  allow  her  to  paste  in 
clips  either  from  the  clipboard  or  a  file.  Other  parts  of  this  second  page  allow  the 
association  of  operations  centers  and  positions  with  the  test  item  and  report  existing  tests 
that  include  this  item.  Examples  of  the  OPS  TEST  Administration  menu,  Test  Item 
Maintenance  query  form,  and  Test  Item  Maintenance  form  are  shown  on  the  next  page. 
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Since  Lt  Jones  passed  his  initial  evaluation  test  so  easily,  Major  Knailem  of  the 
Standardization  and  Evaluation  Branch  decides  he  needs  to  make  the  evaluation  test  for 
that  position  harder.  To  do  so,  he  goes  into  the  Build/Maintain  Tests  functions  from  the 
OPS  TEST  Administration  main  menu.  The  resulting  Build/Maintain  Tests  menu  has  two 
parts.  The  “Master  Tests”  area  allows  Major  Knailem  to  create,  copy,  update,  and  delete 
the  master  tests.  The  “Specific  Tests  for  Crew  Members”  area  allows  him  to  create, 
update,  and  delete  specific  instances  of  Master  Tests  created  for  and  taken  by  individual 
crew  members. 

Major  Knailem  selects  the  test  of  interest  and  clicks  the  “Review/Update  Master  Test” 
button.  OPS  TEST  presents  him  with  a  summary  of  that  test  which  includes  several 
reference  fields  including  the  “Category”  field.  Since  this  field  has  a  value  of  “Eval”  only 
those  with  the  evaluator  version  of  OPS  TEST  Administration  can  access  this  test.  On  this 
summary  form,  there  is  a  line  for  each  item  included  in  the  test  together  with  buttons  to 
view  and  delete  that  particular  item.  A  numbering  field  at  the  beginning  of  each  test  item 
can  be  used  to  specify  a  different  ordering  of  the  items.  To  use  this,  Major  Knailem  would 
just  type  in  a  new  numbering  scheme  and  click  the  “Reorder  Test  Item”  button.  OPS 
TEST  reorders  the  test  items  per  the  new  scheme  and  then  renumbers  them  starting  with 
the  number  one. 

The  “Add  New  Test  Item”  button  brings  up  a  query  form  that  Major  Knailem  can  use  to 
select  new  (harder)  items  for  his  test.  If  he  tries  to  add  an  item  that  is  already  in  his  test, 
OPS  TEST  denies  his  attempt  with  an  appropriate  message.  Other  buttons  on  the  Master 
Test  form  allow  Major  Knailem  to  print  a  summary  of  his  test,  print  a  full  hard  copy  of  the 
test,  and  preview  the  test  pretty  much  as  it  would  appear  for  the  test-taker  on-line. 

In  the  lower  left  corner  of  the  Master  Test  form  are  some  special  fields  primarily  used  for 
training  tests.  The  “Make  on  Request?”  check  box  indicates  that  specific  instances  of  this 
test  do  not  need  to  be  created  for  each  individual  test  taker.  This  is  designed  to  better 
support  the  monthly  training  tests  that  are  not  necessarily  required  of  every  crewmember. 
Checking  the  “Make  on  Request?”  box  activates  the  three  fields  below  it  which  allow  the 
test  administrator  to  specify  if  instances  of  the  test  will  be  randomly  reordered  upon 
creation  and  if  they  will  be  re-entrant.  The  third  field  allows  the  administrator  to  specify  a 
password  to  protect  access  to  the  test. 

Once  Master  Tests  are  created,  the  administrator  can  make  specific  instances  of  that 
Master  Test  for  individual  crewmembers.  This  is  done  via  the  “Specific  Tests  for  Crew 
Members”  functions  on  the  right  side  of  the  Build/Maintain  Tests  form.  These  functions 
make  use  of  the  Specific  Tests  form,  which  provides  several  options  for  Specific  Tests.  An 
example  of  this  form  together  with  the  Build/Maintain  Tests  and  Master  Tests  forms  can  be 
found  on  the  next  page. 

Test  Administrators  use  the  Specific  Test  form  to  create  specific  instances  of  Master  tests 
for  individual  crewmembers.  Such  tests  are  typically  used  for  formal  evaluation  and 
training  tests.  They  can  be  protected  via  individual  passwords  and  a  suspense  may  be 
placed  on  them  to  help  track  tests  that  are  not  yet  taken.  When  the  test-taker  takes  the 
test,  their  start  and  stop  times  and  dates  are  recorded  and  presented  on  this  form  together 
with  their  score.  The  “Preview  On-Line”  button  of  this  form  allows  the  administrator  to  view 
the  test  either  before  or  after  it  is  taken. 
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While  Major  Knailem  has  been  updating  his  evaluation  test,  Capt  Smith  had  taken  a  few 
days  of  leave.  Upon  her  return,  she  wants  to  check  up  on  the  status  of  tests  that  she  had 
prepared  for  crewmembers  prior  to  her  leave.  To  do  this,  she  starts  up  her  copy  of  the 
OPS  TEST  Administration  software  and  goes  into  the  “View  Test  Results”  function.  There 
is  a  simple  query  form  that  she  can  use  to  select  which  test  results  she  wants  to  look  at. 
She  chooses  those  tests  that  she  has  administered  and  that  have  been  taken  but  not  yet 
posted  to  the  test-takers’  records.  By  clicking  the  “View  Selected  Test  Results”  button, 
she  gets  a  summary  of  the  tests  that  have  been  taken  but  not  yet  posted.  Later  she  goes 
back  to  the  query  form  and  looks  for  any  tests  that  she’s  prepared  but  the  crewmember 
trainee  has  not  yet  taken.  For  each  test  presented,  she  can  either  print  the  test  results  or 
view  them  on-line  via  the  Specific  Tests  for  Crew  Members  form,  which  is  the  same  form 
she  used  to  create  that  specific  test. 

While  reviewing  the  test  results,  she  wonders  if  there  are  some  items  out  there  that  are 
just  not  working  as  well  as  she  would  like.  To  address  this  curiosity,  she  calls  up  the  “Test 
Item  Analysis”  function  from  the  main  OPS  TEST  Administration  menu.  Again,  she  gets  a 
query  form  that  she  can  use  to  select  the  test  items  that  she  would  like  to  analyze.  She 
specifies  that  she  wants  items  that  have  an  overall  average  (ease)  of  less  than  60%  and 
clicks  the  “Perform  Test  Item  Analysis”  button.  She’s  warned  that  it  may  take  a  few 
minutes  to  complete  the  analysis  and  then  she  gets  a  form  that  shows  the  results  for  the 
first  of  the  selected  test  items.  The  form  presents  the  overall  ease  of  the  item  and  shows  a 
distribution  of  the  test  responses  that  reports  which  alternatives  are  being  chosen  most 
often.  The  same  information  is  also  shown  grouped  by  crew  positions  and  by  individual 
test-takers.  With  this  information  she  is  able  to  identify  test  items  that  may  have  awkward 
or  misleading  wording. 

On  the  next  page  you’ll  find  examples  of  the  screens  that  Capt  Smith  saw  as  she 
accomplished  these  tasks. 

At  their  next  Operational  Readiness  Inspection,  the  99th  Space  Squadron  scores  an 
Outstanding  and  the  inspectors  note  that  the  use  of  OPS  TEST  to  support  formal 
evaluation  and  training  testing  as  well  as  crewmember  practice  testing  has  helped  to 
significantly  raise  crew  performance  while  reducing  the  administrative  burden  on  the 
training  and  stan/eval  functions. 


11 


Select  Specific  Tesl  Results  To  View 


(Specify  any  desired  test  results  selection  criteria  and  click  'View  Selected  T est  Results.) 
Test  Status 
C"  Any 

Completed,  But  Not  Yet  Posted 
(*  Not  Yet  Taken 


Completed  After 


Master  Test  [~ 

T  est  T  aker  |" 

Administrator  [Smith 
Position  1 

Before  | 


View  Selected  T est  Results 


*l 


View  'fi'  Tesl  Results 


Test  ID 

Crew  Member  Tested 

Crew  Position 

Completed  Suspense  Score 

Posted? 

► 

[Auringer,  Amos  A  (E-6) 

[MWC/CCH 

|  1 5-Dec-99  | 

I~  Print  Results  |  View  Details  | 

[Missile  Warning 

[Hadfield,  Steve  M  (O-S) 

[MWOCDR 

|  15-Nov-99  | 

I-  Print  Results  |  View  Details  | 

[Missile  Warning 

|Revak,  Marie  A  (0-4) 

[MWCiCDR 

|  01 -Dec-99  |~ 

r~  Print  Results  |  ViewOetails  | 

BE  Select  Test  Items  for  Analysis 


m 


Task 

References 


Enter  your  selection  criteria  and  click  on  the  Perform  Test  Item  Analysis  button. 
I - 3  Category  |  3]  Knowledge  [ 


Item  Numbers  --  from:  [ 
Test  Results --from:  |" 
Number  of  Misses  in:  [" 
Overall  Ease  in: 


r 


r 

r 

Master  Test 


Center 

Position 


~n 


Test  Item  Analysis 


Item#  Answer  Ease  Total 

D  |  33.3%  [  3 


A  B  C  (T]  E 


Results  mcrew  Position 


l  so  o%  |  2  |  I  |  r  |  r  |  * 


1  »  l>l  l>  ■  I  of  2 


Results  History  mate  Completed.  Master  Test,  Crew  Position,  ItomeolTesl-Taker.  Ttielr  Response] 


> 

|  03-NOV-99  [Missile  Warning 

[CCAIO 

[Hadfield,  Steve  M  (0-5) 

lc 

- 

|  17-Apr-99  [Multimedia  Formal 

[CC/CD 

[Hadfield,  Steve  M  (0-5) 

lc 

|  17-Apr-99  [Multimedia  Formal 

[CC/CD 

[Hadfield,  Steve  M  (0-5) 

P 

Record:  H  |  <  1 1  1  ►  1  M 

Jr-  1  of  3 

zi 

Record:  H  I  <  I  [ 


12 


History  of  OPS  TESTs  Development 

How  did  OPS  TEST  come  about  and  what  was  learned  from  its  first 
use? 


This  section  of  the  report  provides  a  brief  summary  of  how  OPS  TEST  was  initially 
developed.  It  also  discusses  some  key  lessons  learned  during  the  development  and  initial 
implementation.  These  may  have  relevance  to  those  considering  adoption  of  OPS  TEST 
for  their  unit. 


Story  behind  OPS  TEST’S  initial  development 


The  idea  for  an  automated  testbank  and  integrated  on-line  testing  capability  originally 
came  from  Lt  Col  Ray  Yelle,  USAF,  while  he  was  the  Chief  of  the 
Standardization/Evaluation  (stan/eval)  branch  in  the  US  Space  Command’s  Cheyenne 
Mountain  Operations  Center  (CMOC).  He  had  heard  of  the  an  automated  testbank 
developed  and  used  by  the  US  Air  Force  Academy’s  Department  of  Mathematical 
Sciences  and  came  with  his  staff  to  the  Academy  for  a  demonstration.  This  initiated  a 
collaborative  effort  to  design  and  develop  an  automated  testbank  and  on-line  test-taking 
capability  to  support  both  the  stan/eval  and  training  functions  within  CMOC.  MSgt  Amos 
Auringer  from  CMOC  led  the  requirements  definition  effort  while  Lt  Col  Steve  Hadfield 
developed  the  software.  The  capability  developed  was  known  as  the  CMOC  Testbank 
and  received  extensive  support  from  the  US  Air  Force  Academy’s  newly  formed  Institute 
for  Information  Technology  Applications  (IITA). 

The  enthusiastic  acceptance  of  the  CMOC  Testbank  by  both  the  stan/eval  and  training 
personnel  in  CMOC  as  well  as  the  crewmembers  provided  evidence  that  the  CMOC 
Testbank  could  be  of  significant  benefit  to  other  operational  DoD  units.  Under  IITA 
sponsorship,  the  CMOC-unique  parts  of  the  testbank  application  were  parameterized  so 
they  could  be  easily  modified  for  other  users.  The  most  significant  of  these  CMOC- 
specific  features  was  a  link  to  the  Operational  Decision  Support  System  (ODESSY)  also 
developed  for  CMOC  under  the  initiatives  of  Lt  Col  Yelle  and  MSgt  Auringer.  The 
ODESSY  system  tracks  all  personnel,  training,  evaluation,  and  crew  information  and 
provides  sophisticated  crew  scheduling  features.  Initially,  the  CMOC  Testbank  would  tap 
into  the  ODESSY  to  obtain  information  on  crewmembers,  task  codings  (originally  called 
Performance  Evaluation  Criteria  (PEC)),  and  center/crew  position  information.  As  the 
ODESSY  and  CMOC  Testbank  were  refined,  differences  in  their  security  models  drove  a 
separation  of  the  two  systems.  Instead  of  the  CMOC  Testbank  directly  accessing  the 
ODESSY  data,  we  chose  to  periodically  update  the  testbank  with  copies  of  the  data  it 
needed.  For  use  at  other  units,  we  added  forms  to  the  testbank  administration  component 
so  that  the  personnel,  task  coding,  and  center/crew  positions  information  could  be  directly 
maintained  from  within  the  testbank  administration  functions.  With  this  additional 
capability,  the  testbank  application  became  a  fully  self-sufficient  application  and  we 
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renamed  it  to  the  “Operational  Test  and  Evaluation  Support  Tool”,  abbreviated  as  “OPS 
TEST”. 

In  addition  to  the  development  of  the  OPS  TEST  software,  the  collaboration  between 
CMOC  and  USAFA  included  a  series  of  very  well  received  workshops  on  effective  test 
item  development  by  Major  Marie  Revak  of  the  US  Air  Force  Academy’s  Center  for 
Educational  Excellence.  Due  to  personnel  turnover  in  the  CMOC  training  and  stan/eval 
teams,  the  workshops  have  been  given  every  6-12  months  to  assist  in  preparing  new 
team  members  to  create  effective  test  items.  In  addition  to  the  test  item  development 
workshops,  we  also  provided  training  on  the  testbank  administration  functions  and  Major 
Revak  conducted  a  comprehensive  review  of  the  existing  test  items  providing  suggestions 
on  how  a  number  of  them  could  be  improved. 


Lessons  learned  from  OPS  TEST  initial  implementation 


During  the  initial  development  and  implementation  of  what  is  now  OPS  TEST,  there  were 
several  lessons  learned  that  may  be  of  significance  to  other  operational  units  considering 
the  adoption  of  OPS  TEST.  First  among  these  were  the  issues  resulting  from  the 
migration  from  the  paper-based  testing  to  the  on-line  testing  provided  by  OPS  TEST.  For 
the  stan/eval  group  this  involved  the  conversion  of  their  tests  from  Microsoft  Excel 
spreadsheets  to  Microsoft  Word  documents  that  were  then  incorporated  into  the  OPS 
TEST  database.  Because  a  consistent  structure  was  used  for  the  Excel-based  tests,  we 
were  able  to  write  some  software  to  convert  the  existing  test  items  automatically  into  Word 
documents  and  import  them  into  the  OPS  TEST  database.  However,  we  did  need  to  take 
care  to  insure  that  all  the  items  were  properly  converted  and  there  were  a  significant 
number  of  duplicate  test  items  created  as  some  test  items  were  used  in  multiple  tests. 
Identification  and  elimination  of  duplicate  test  items  was  a  significant  effort. 

Migration  of  the  training  test  items  was  more  labor-intensive.  They  were  originally  held  as 
Word  documents  which  was  nice  in  that  they  could  be  cut  and  pasted  directly  into  the 
testbank  database,  but  there  was  no  easily  recognizable  structure  to  them  so  this  process 
could  not  be  easily  automated.  However,  the  cut  and  paste  capabilities  of  Windows  made 
this  process  fairly  straight  forward  and  it  typically  took  a  minute  or  two  per  test  item  to 
complete  the  migration. 

Another  key  issue  from  the  initial  implementation  was  the  integration  of  the  testbank 
application  with  the  ODESSY  decision  support  database.  Some  of  the  problems  with  this 
integration  occurred  because  the  testbank  and  the  ODESSY  were  developed  by 
geographically  separated  developers,  but  these  issues  were  fairly  easily  resolved  by 
periodic  meetings  of  the  two  developers  and  the  CMOC  coordinator,  MSgt  Auringer.  The 
bigger  problem  in  the  integration  dealt  with  the  different  security  models  required  of  the 
two  applications.  The  ODESSY  required  user-level  security  where  groups  of  users 
required  different  types  of  access  to  different  subsets  of  data.  The  testbank  application 
required  a  highly  controlled  access  to  the  data  (test  items)  with  the  access  controlled  by 
both  time  and  user  type.  As  a  result,  the  two  applications  used  different  security 
mechanisms  within  the  Microsoft  Access  database  management  software  and  there  were 
compatibility  issues  between  the  two  mechanisms.  The  most  efficient  and  effective 
solution  was  to  separate  the  two  applications. 

Other  lessons  learned  deal  with  new  releases  of  the  Microsoft  Access  database 
management  software  and,  to  a  much  lesser  degree,  the  Microsoft  Office  suite  of 
programs  in  general.  The  testbank  application  was  initially  developed  in  Access  97  at  the 
US  Air  Force  Academy,  but  CMOC  was  using  Access  95.  Several  key  features  of  Access 
97  were  needed,  especially  for  the  automated  migration  of  the  stan/eval  test  items. 
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Access  applications  are  typically  not  backward  compatible  to  previous  versions  of  Access 
so  CMOC  had  to  upgrade  to  Access  97  or  we  needed  to  load  a  run-time  version  of  Access 
97  with  the  testbank  application.  Use  of  a  run-time  version  of  Access  on  a  computer  that 
has  some  other  version  of  Access  can  easily  result  in  problems  if  not  properly  done  and 
should  be  avoided  if  possible.  We  are  currently  faced  with  similar  problems  as  Access 
2000  and  Office  2000  are  now  available.  With  each  new  release  of  Access  and  Office,  the 
OPS  TEST  software  will  need  to  be  migrated  and  there  will  be  support  and  compatibility 
issues  with  the  earlier  versions  of  these  programs. 

Finally,  there  are  some  minor  configuration  issues  as  OPS  TEST  becomes  more  widely 
used.  Both  the  test-taking  and  administration  components  of  OPS  TEST  access  the 
common  backend  database  of  test  items,  tests,  and  results  that  reside  on  a  file  server  on 
the  local  area  network  (LAN).  This  backend  database  is  encrypted  and  password- 
protected  so  the  test-taking  and  administration  components  need  to  know  the  path  to  the 
backend  database  and  its  password.  Furthermore,  these  will  vary  with  each 
implementation  and  may  need  to  be  updated  from  time  to  time  for  a  particular 
implementation.  The  OPS  TEST  software  automatically  remembers  the  location  and 
password  of  the  backend  database  from  the  last  access.  If  either  the  location  or  password 
has  changes,  the  OPS  TEST  software  will  detect  the  problem  and  ask  the  user  for  the 
new  location  and  password.  Furthermore,  OPS  TEST  administrators  can  freely  distribute 
the  password  to  users  since  the  password  only  works  in  conjunction  with  the  OPS  TEST 
software  and  not  directly  with  the  Access  database  management  system. 
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Details  of  OPS  TESFs  Capabilities 

What  exactly  can  OPS  TEST  provide? 


In  this  section  of  the  report,  we  provide  a  detailed  technical  summary  of  OPS  TEST’S 
capabilities  and  features.  It  is  intended  to  provide  both  a  comprehensive  overview  and 
reference  for  the  implementation  decision-maker. 


The  OPS  TEST  database 


■  Holds  test  items  as  Microsoft  Word  documents  which  facilitates  embedding  of 
pictures,  graphs,  charts,  and  clip  art  within  test  items. 

■  Includes  reference  data  with  each  test  item  for  use  in  retrieval  and  evaluation  of  test 
items.  This  data  includes: 

■  Task  code 

■  Security  classification 

■  Date  created 

■  Date  last  reviewed 

■  Instructional  Systems  Design  (ISD)  knowledge/proficiency  level  code 

■  Number  of  answer  choices 

■  Designation  of  the  correct  answer 

■  Reference  to  the  governing  regulation,  operating  instruction  or  other  guidance 

■  Category  of  the  item  (training,  evaluation,  or  Master  Question  File) 

■  Feedback  to  provide  to  test-takers  when  the  test  item  is  answered  incorrectly 

■  Hyperlink  to  associated  additional  feedback  or  reference  information 

■  Optional  multimedia  clip  to  be  played  as  part  of  the  test  item 

■  Listing  and  count  of  the  number  of  tests  the  item  is  used  in 

■  Operational  centers  and  positions  associated  with  the  test  item 
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■  Contains  “master  tests”  typically  based  on  a  particular  crew  position.  These  master 
tests  include  the  items  mentioned  below  and  the  individual  test  items  that  are  linked 
into  the  test. 

■  Master  test’s  name 

■  Associated  operational  center  and  position 

■  Category  (typically  evaluation,  training,  or  Master  Question  File) 

■  Type  of  test  (initial,  recurring,  or  special) 

■  Office  of  primary  responsibility 

■  The  individual  that  created  the  test 

■  Additional  optional  parameters  allow  the  test  to  be  created  on  demand  by  the  test- 
taker  (as  opposed  to  individually  created  for  a  specific  test-takers  by  a  test 
administrator),  automatic  random  reordering  upon  creation,  set  with  permissions 
to  exit  and  re-enter  the  test,  and  password  protected. 

■  Includes  “specific  tests”  created  by  test  administrators  from  master  tests  for  specific 
individual  evaluations.  Specific  test  information  includes: 

■  Name  of  the  master  test  from  which  each  specific  test  was  built 

■  Individual  test-taker  that  the  specific  test  was  built  for 

■  Password  used  to  protect  access  to  the  test 

■  Operational  center  and  position  for  the  test 

■  Whether  or  not  the  test  can  be  re-entered 

■  Date  created 

■  Whether  it’s  a  retest 

■  Suspense  date  by  which  the  test  must  be  taken 

■  Administrator  that  built  the  test 

■  Start  and  finish  dates  and  times  from  when  the  test-taker  took  the  test 

■  Resulting  score 

■  Whether  the  test  results  have  been  posted  to  the  test-taker’s  records 

■  Additionally  the  database  includes  information  on  the  personnel  assigned  to  the  crews 
as  well  as  the  trainers  and  evaluators,  a  listing  of  the  task  codes,  and  the  operational 
center  and  position  designators.  This  information  can  be  imported  from  an 
accompanying  database  such  as  the  ODESSY  (Operational  Decision  Support 
System)  in  the  Cheyenne  Mountain  Operations  Center  or  it  can  be  maintained  via 
forms  under  the  Configuration  menu  in  the  OPS  TEST  Administration  component. 
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Test-taking  capabilities 


■  Take  formal  evaluations  tests  that  are  specifically  created  for  individuals  by  a  test 
administrator  with  password-protected  access. 

■  An  option  for  re-entry  can  be  set  by  the  test  administrator. 

■  Navigation  buttons  allow  the  test-taker  to  move  forward  or  backward  within  the 
test  either  an  item  at  a  time  or  to  the  next  (or  previous)  unanswered  test  item. 

■  Warnings  and  recovery  options  are  provided  if  test-taker  has  left  any  items 
unanswered  when  they  submit  the  test  for  grading. 

■  Upon  grading,  the  test-taker  receives  a  report  of  their  score  and  has  the  option  to 
review  any  or  all  of  the  test  items  that  they  missed  along  with  the  feedback  text 
associated  with  the  item,  and  any  associated  hyperlinks  to  additional  reference 
information. 

■  Dynamically  create  and  take  monthly  training  tests,  which  may  optionally  be  randomly 
ordered,  re-entrant,  and/or  password-protected.  Mechanisms  for  taking  these  tests 
are  identical  to  those  employed  for  formal  evaluation  tests. 

■  Access  the  Master  Question  File,  which  is  a  repository  of  “practice”  test  items  that 
crewmembers  can  use  for  self-evaluation. 

■  Test  items  from  the  Master  Question  File  can  be  selected  via  any  combination  of 
a  number  of  different  criteria  including  task  code,  Instructional  Systems 
Development  knowledge/proficiency  level,  operational  center,  crew  position,  and 
governing  reference. 

■  When  responding  to  test  items  from  the  Master  Question  File,  the  test-taker  is 
given  immediate  feedback  as  each  item  is  answered. 

■  The  test-taker  may  exit  a  set  of  selected  test  items  at  any  time  and  their  current 
percentage  of  correct  answers  is  reported  to  them. 

NOTE:  All  test-taking  automatically  checks  for  the  presence  of  associated  multimedia  clips  and  provides  a  button 
with  which  the  test-taker  can  view  the  clip  as  many  times  as  they  like.  The  only  qualification  is  that  the  test-taker’s 
computer  must  have  a  program  and  association  setup  to  view  that  particular  type  of  multimedia.  Similarly,  the 
presence  of  hyperlinks  to  associated  reference  material  for  feedback  is  automatically  detected  and  presented,  if 
available,  but  requires  associated  software  to  view  that  type  of  material. 
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Test  administration  capabilities 


■  Add,  modify,  delete,  and  selectively  locate  individual  test  items  and  their  reference 
information. 

■  Build,  update,  copy,  and  delete  both  master  and  specific  tests.  Test  items  for 
inclusion  in  master  tests  may  be  searched  for  and  selected  based  on  any  combination 
of  the  selection  criteria.  Checks  are  made  to  preclude  redundant  use  of  specific  test 
items.  Selected  test  items  shown  for  consideration  are  displayed  with  all  the 
associated  feedback  information  (including  hyperlinks  to  any  associated  references) 
and  any  accompanying  multimedia  clips.  Specific  tests  may  be  previewed,  randomly 
reordered,  updated,  or  reset  to  allow  re-entry  by  the  test-taker.  While  most  testing  is 
conducted  on-line,  the  ability  exists  to  produce  printed  tests  and  their  answer  keys. 

■  Obtain  test  results  and  status  reports  via  a  number  different  criteria  including  tests 

completed  but  not  yet  posted,  tests  waiting  to  be  taken,  or  any  test  together  with 

selection  criteria  based  on  the  master  test,  test-taker,  test  administrator,  crew  position, 
and  date  range  of  when  the  test  was  taken.  Test  results  can  be  viewed  and/or  printed 
in  either  summary  form  or  as  a  full  report  by  test  item. 

■  Produce  performance  analysis  reports  for  individual  test  items  via  a  number  of 

different  selection  criteria.  Both  ease  (percentage  of  test-takers  that  correctly 

answered  the  test  item)  and  distracter  selection  are  reported  for  overall  usage,  usage 
by  crew  position,  as  well  as  by  individual  test-takers. 

NOTE:  The  query  forms  used  to  specify  search  criteria  for  locating  specific  test  items  or  results  have  pull-down 
lists  of  options  for  most  selection  criteria  fields.  These  options  are  built  based  on  the  data  that  has  been  entered 
into  the  database,  thus,  they  are  “self-adapting”  for  new  implementations.  Furthermore,  many  of  these  selection 
fields  also  allow  the  use  of  “wildcards”  to  generalize  the  search  parameters. 
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OPS  TEST  configuration  settings 


■  Personal  computers  running  Windows  95/98/NT/2000  with  Microsoft  Word  installed 
are  needed  to  run  OPS  TEST.  Preferably  they  also  have  Microsoft  Access  97 
together  with  programs  and  associations  set  to  view  whatever  hyperlinked  feedback 
references  and/or  multimedia  clips  that  may  be  included  with  the  test  items.  If  Access 
97  is  not  available,  an  installation  set  up  can  be  created  that  automatically  loads  a 
distribute-able  run-time  version  of  Access.  Typically,  an  Intel  Pentium-class  machine 
with  at  least  16  MBytes  of  RAM  and  3  MB  of  disk  space  is  sufficient  to  host  the  OPS 
TEST  Administration  component  and  1  MB  of  disk  space  for  the  OPS  TEST  Test- 
Taking  component. 

■  OPS  TEST  typically  uses  a  local  area  network  (LAN)  running  at  an  appropriate 
“system  high”  security  level  for  the  test  items.  The  LAN  must  have  a  network  disk 
drive  accessible  by  all  test  administrators  and  test-takers  with  enough  capacity  to 
accommodate  the  testbank  database.  Typically,  the  disk  space  requirement  is  less 
than  10,000  bytes  per  test  item,  but  test  items  with  embedded  graphics  or  associated 
multimedia  can  greatly  increase  the  size  of  test  items. 
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Chapter 


Support  for  OPS  TEST  Implementations 

What  support  can  a  unit  expect  if  OPS  TEST  is  adopted?  What 
does  the  adopting  unit  need  to  provide? 


IITA  support  provided  for  OPS  TEST  implementations 


If  your  unit  decides  to  adopt  OPS  TEST  to  support  your  operational  training  and/or 
standardization/evaluation  functions,  the  Institute  for  Information  Technology  Applications 
(IITA)  can  provide  some  key  support  to  assist  you.  Specifically,  IITA  is  prepared  to 
provide: 

■  Master  and  backup  compact  disks  that  include  the  setup  programs  for  both  the  OPS 
TEST  Administration  and  Test-Taking  software  components,  an  empty  database,  and 
electronic  copies  of  both  the  OPS  TEST  Administration  and  Test-Taking  manuals. 
Upon  request,  IITA  can  include  separate  versions  of  the  OPS  TEST  Administration 
components  for  the  training  and  standardization/evaluation  functions  where  the 
training  version  is  precluded  from  accessing  tests  or  test  items  in  the  “Eval”  category. 

■  On-site  installation  and  initial  training  support  by  one  IITA  technical  support  person. 
This  support  would  typically  include  a  2-3  day  TDY  funded  by  the  adopting  unit. 

■  Several  hardcopies  of  the  OPS  TEST  Administration  and  Test-Taking  user  manuals 
together  with  PowerPoint  training  slides  and  electronic  versions  of  the  same. 

■  Updated  version  releases  of  the  OPS  TEST  Administration  and  Test-Taking  software 
to  keep  OPS  TEST  current  with  subsequent  releases  of  Microsoft  Office  (specifically 
Word  and  Access).  Software  releases  may  also  include  problem  resolutions  and 
functional  enhancements  possibly  including  items  identified  by  the  adopting  unit. 
However,  IITA  will  serve  as  the  final  software  configuration  management  authority  and 
will  make  all  decisions  regarding  what  is  and  is  not  included  in  subsequent  software 
releases. 

■  Review  of  current  test  items  and  training  on  effective  test  item  development.  This 
support  would  also  be  funded  by  the  adopting  unit  and  could  be  accomplished  in 
conjunction  with  the  initial  installation. 

■  Limited  telephone  technical  support  for  user  and  configuration  issues  based  on  the 
availability  of  IITA  technical  support  personnel.  (However,  IITA  cannot  guarantee 
such  availability). 

IITA  will  provide  the  support  mentioned  above  until  at  least  the  summer  of  2001.  At  this 
time,  IITA  will  evaluate  the  extent  of  OPS  TEST  usage  and  formulate  continuing  support 
plans  as  they  deem  appropriate. 
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Requirements  of  the  adopting  unit 


The  operational  unit  adopting  OPS  TEST  needs  to  provide  certain  support  required  of  an 

OPS  TEST  installation.  This  adopting  unit  support  must  include: 

■  Local  area  network  with  support  for  a  system  high  level  of  security  for  classified 
material  together  with  a  network  file  server  capable  of  hosting  the  backend  database. 
A  typical  database  will  require  1-100MB  of  storage  per  every  100  test  items 
depending  on  how  many  pictures,  graphs,  and  multimedia  clips  are  associated  with 
the  test  items. 

■  Information  concerning  the  configuration  of  their  local  area  network  and  shared 
network  file  server  that  will  host  the  OPS  TEST  backend  database.  The  primary 
requirement  here  is  a  path  to  the  location  of  the  backend  database  that  will  be  need  to 
configure  each  installation  of  the  OPS  TEST  administration  and  test-taking  software. 

■  Installations  of  the  Microsoft  Windows  NT/95/98/2000,  Word,  and  Access  (version  97 
or  newer)  on  any  machines  hosting  OPS  TEST.  We  can  work-around  the  absence  of 
Access  using  a  distributable,  run-time  version  of  Access  that  we  can  build  into  your 
installation. 

■  Funding  for  any  TDYs  required  for  IITA  technical  support  personnel  to  assist  with  the 
initial  installation/configuration  of  the  OPS  TEST  software  and/or  IITA  training 
personnel. 

■  On-going  training  of  new  personnel  on  the  use  of  OPS  TEST. 

■  Reproduction  of  training  materials  and  user  manuals  beyond  the  limited  initial  set 
provided  by  IITA. 

■  Operational  use  of  OPS  TEST  including  maintenance  of  tests  and  test  items. 

■  Software  maintenance  beyond  that  specified  as  provided  by  IITA  in  the  previous 
section. 
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About  the  Institute 


The  Institute  for  Information  Technology  Applications  (IITA)  was  formed  in  1998  to  provide 
a  means  to  research  and  investigate  new  applications  of  information  technology.  The 
Institute  encourages  research  in  education  and  applications  of  the  technology  to  Air  Force 
problems  that  have  a  policy,  management,  or  military  importance.  Research  grants 
enhance  professional  development  of  researchers  by  providing  opportunities  to  work  on 
actual  problems  and  to  develop  a  professional  network. 

Sponsorship  for  the  Institute  is  provided  by  the  Secretary  of  the  Air  Force  for  Acquisition, 
the  Air  Force  Office  of  Scientific  Research,  and  the  Dean  of  Faculty  at  the  U.S.  Air  Force 
Academy.  IITA  coordinates  a  multidisciplinary  approach  to  research  that  incorporates  a 
wide  variety  of  skills  with  cost-effective  methods  to  achieve  significant  results.  Proposals 
from  the  military  and  academic  communities  may  be  submitted  at  any  time  since  awards 
are  made  on  a  rolling  basis.  Researchers  have  access  to  a  highly  flexible  laboratory  with 
broad  bandwidth  and  diverse  computing  platforms. 

To  explore  multifaceted  topics,  the  Institute  hosts  single-theme  conferences  to  encourage 
debate  and  discussion  on  issues  facing  the  academic  and  military  components  of  the 
nation.  More  narrowly  focused  workshops  encourage  policy  discussion  and  potential 
solutions.  IITA  distributes  conference  proceedings  and  other  publications  nation-wide  to 
those  interested  or  affected  by  the  subject  matter. 


Comments  pertaining  to  this  report  are  invited  and  should  be  directed  to: 
Sharon  Richardson 

Director  of  Conferences  and  Publications 

Institute  for  Information  Technology  Applications 

HQ  USAFA/DFPS 

2354  Fairchild  Drive,  Suite  6L16D 

USAF  Academy  CO  80840-6258 

Tel.  (719)  333-2746;  Fax  (719)  333-2945 

e-mail:  sharon.richardson@usafa.af.mil 
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